The instant invention relates to a method of detecting cloud formations and to a method of detecting and avoiding cloud formations when Visual Meteorological Conditions (VMC) must be observed.
Unmanned Aircraft Systems (UAS) and Remotely Piloted Aircraft (RPA) are becoming indispensable tools for a huge number of aeronautical tasks such as data collection and wide area surveillance. Currently, however, UAS/RPA face limitations on their utilization in civil airspace because they do not have the capability to sense and avoid (SAA) other air traffic and clouds. Passive electro-optical (EO) or infra-red (IR) cameras can be used to provide the “sense” capability. The small size, weight and power characteristics of EO/IR sensors make them uniquely attractive for all sizes of UAS/RPA. An aircraft dependent on EO/IR sensors to detect other air traffic must also follow the Federal Aviation Administration's (FAA) Visual Flight Rules (VFR) rules which prohibit the operation of any aircraft under conditions when flight visibility is less, or distance to clouds, is less than that prescribed for the corresponding altitude and class of airspace.
This disclosure provides a novel approach that endows cloud detection and avoidance capabilities to UAS/RPA based on passive EO/IR sensors. Specifically, the proposed approach processes image sequences from a monocular camera mounted on a UAS/RPA and performs cloud detection, 3D reconstruction of cloud formations and avoidance of clouds per FAA mandates. In what follows, the moving platform carrying the EO/IR sensor is referred to as own-ship and a dynamically moving aircraft sharing the same airspace as the own-ship is referred to as an intruder.
International air navigation regulations require all aircraft, including UAS/RPA, to maintain radar separation from all other traffic. UAS/RPA equipped with radar can meet this requirement. Although radar is anticipated to serve as the primary sensor for Sense and Avoid (SAA) in many UAS/RPA, there are circumstances where radio frequency emission is undesirable. Furthermore, some air platforms may have payload size, weight and power constraints which may preclude the use of an appropriate radar sensor system. In such situations, UAS/RPA is required to maintain operations in Visual Meteorological Conditions (VMC), that is, it will have to stay clear of clouds so that it can see other air traffic using on-board EO/IR sensors. The EO/IR sensors may be the same ones used for SAA. Additionally, a large number of UAS/RPA are already equipped with EO/IR cameras used for ISR (Intelligence, Surveillance and Reconnaissance) and other missions which can be used to fulfill the dual purpose of cloud detection and avoidance. The main problem addressed by the proposed invention is to exploit image sequences from such passive EO/IR sensors for autonomous cloud detection and avoidance. Specifically, the above problem involves processing monocular image sequences, extracting 3D cloud information using known own-ship position, attitude, intrinsic and extrinsic camera calibration information and planning paths to successfully steer clear of clouds, thereby ensuring autonomous operations in VMC.
The inventions described in this disclosure can be categorized as belonging to two research areas, namely real-time 3D reconstruction of cloud formations from cameras and path planning for cloud avoidance.
Previous research on 3D cloud reconstruction from cameras has focused on cloud modeling using more sophisticated sensors than we have at our disposal. Some prior art has been able to create volumetric cloud estimates using a color stereo camera. However, there are 2 limitations with this approach. Firstly, the 2 cameras constituting the stereo pair are assumed to be static in 3D. Secondly, the 2 cameras have a large baseline of 640 meters between them. Both of these assumptions are not valid for UAS/RPA applications. Also, currently many UAS/RPA have access to monocular grayscale cameras only.
Other work has created realistic maps of cloud locations in the sky, but these techniques are ground-based and only generate 2D estimates of cloud locations.
Yet another body of work rendered a cloud from any perspective angle, even though the input data is 2D. This work is related to our work in that it is able to recreate realistic cloud structures however their methods rely on orthographic satellite images. The invention described in this disclosure, in general, can be adapted to handle grayscale, color, infra-red, monocular, stereo and multiple camera configurations.
In the present method of cloud avoidance, following the cloud reconstruction, one needs to check if the nominal path of the UAS/RPA is passing through the clouds. If the nominal UAS/RPA path is passing through the clouds, the path planner will compute another collision free path for the UAS/RPA. Path planning for collision avoidance of UAS/RPA is an active area of research and typically it involves two steps. Firstly, a discrete path connecting the starting point and the final goal is constructed while avoiding obstacles. For this step, methods like probabilistic roadmaps or Rapidly Exploring Random Trees (RRT) can be used. This connected path consists of discretized line segments and arcs that the UAS/RPA can execute. The second step is the trajectory smoothing step, in which the discrete connected paths from the first step are smoothed so that the UAS/RPA can execute it. For example, RRT has been previously utilized for path planning followed by a trajectory smoothing routine using cubic Bezier spiral curves.
A two-phase method for motion planning of fixed wing UAS/RPA in 3D environments has also been implemented. In this scenario, a coarse global planner first computes a kinematically feasible obstacle free path in a discretized 3D workspace which roughly satisfies the kinematic constraints of the UAS/RPA. Given a coarse global path, a fine local motion planner is used to compute a more accurate trajectory for the UAS/RPA at a higher level of detail. The local planner is iterated as the UAS/RPA traverses and refines the global path as needed up to its planning horizon. This work is related to the present disclosure in that the kinodynamic constraints of the UAS/RPA are accounted for in the path generation stage. However, in addition to accounting for the kinodynamic constraints, the present method is designed for the UAS/RPA to operate in National Air Space (NAS). We designed the algorithm so that the UAS/RPA:
1. Adheres to the cloud separation constraints imposed by FAA;
2. Incorporates the air corridor constraints; and
3. Avoids clouds in Real-Time.
The present disclosure also develops a new way to represent the obstacles, such as clouds or other static objects that make them more amenable for path planning and trajectory smoothing.
The inventions described in this document are concerned with processing image sequences from monocular cameras mounted on UAS/RPA, real-time extraction of 3D cloud formation information and planning paths in 3D to avoid such formations. The invention can also be used for processing color image sequences from stereo or multiple cameras. In this disclosure, we present a specific case involving a grayscale monocular camera.
Given below is a summary of inventions:
1. A novel monocular EO/IR camera based system design, comprising autonomous and intelligent cloud detection and avoidance algorithms, thereby enabling the UAS/RPA to remain in VMC which is a key-component of maintaining due-regard for the safety of other air traffic in the airspace. The design is scalable to number of color channels & cameras available on-board e.g. using stereo color instead of monocular EO/IR camera.
2. A novel incremental method (optimal in least squares sense) for extracting 3D cloud information from monocular image sequences. A sparse set of feature points are detected and tracked in the image sequence. An incremental approach processes each 2D feature track and estimates corresponding 3D positions. It is shown that as more 2D measurements are available, 3D estimates can be incrementally updated in constant time, independent of 2D track length. Each 2D track is treated independently of others in the 3D reconstruction process. If available, additional 2D measurements from EO/IR stereo or more cameras can be easily incorporated to improve 3D reconstruction. This method is easily scalable to number of cameras available on-board.
3. A novel probabilistic and incremental method for estimating uncertainties in 3D feature point estimates. Uncertainties are used as a measure to discard unreliable and irrelevant 3D feature tracks. Modeling the noise in 2D feature tracks and 3D camera positions as Gaussian distributions, it is shown that the uncertainty in 3D feature estimate can be incrementally updated in constant time, independent of track length using linearized error propagation techniques.
4. A novel method for fast and hybrid clustering of reliable 3D feature tracks into clusters based on 3D proximity and image appearance similarity information. Such clusters are considered to represent 3D cloud formations and can be output to a display or input to a path planning algorithm.
5. A novel method to autonomously generate way points to avoid clouds while following FAA VFR for cloud avoidance.
6. A novel framework to jointly avoid clouds as well as other air traffic that may be present in the vicinity.
Other objects, features and advantages of the invention shall become apparent as the description thereof proceeds when considered in connection with the accompanying illustrative drawings.